How one can Debug AI Coding Brokers When They Change the Flawed Factor

agent a request, the arduous half just isn’t at all times writing code. The arduous half is displaying that the agent understood the request, modified the suitable place, and verified the suitable outcome.

An agent can do a number of issues throughout a run and nonetheless finish with a brief reply like “mounted.” That reply hides the essential particulars: what it inspected, what it modified, what it ran, and what proved the repair labored. With out proof, you can’t inform whether or not the ultimate test matched the person’s authentic grievance.

That hole reveals up in odd coding requests. A passing construct can present the app nonetheless compiles, regardless that the button remains to be damaged. A passing take a look at can imply the assertion bought weaker. A believable route change can nonetheless miss the handler that serves the response.

A coding agent ought to change code and show what it inspected, what it modified, what it ran, and what outcome made the repair acceptable.

Software program engineering already has a reputation for a part of this downside: traceability. Torkar et al. describe necessities traceability as the flexibility to comply with a requirement from its emergence to its success. In coding agent work, the identical thought turns into extra instant: are you able to join the person’s request to the information inspected, the patch utilized, the command outcome, and the ultimate verification test?

Mannequin Context Protocol (MCP) describes a typical approach for synthetic intelligence functions to hook up with information, instruments, information sources, and different exterior programs. That commonplace solely helps if the instrument run leaves proof a developer can assessment.

The tutorial tackles an issue each builders and vibe coders run into: the agent adjustments one thing, sounds assured, and leaves you guessing whether or not it touched the suitable a part of the app.

The instance makes use of a person interface bug as a result of visible fixes are simple to get mistaken and simple to confirm in a browser. A pricing card has a main button that overflows on a cellular display screen measurement. The restore is a focused Cascading Model Sheets (CSS) change. The helpful half is the saved proof across the restore: file reads, a mistaken selector try caught by the browser test, a focused patch, construct output, Doc Object Mannequin (DOM) measurements, and earlier than and after screenshots.

By the tip, it’s best to have a assessment sample you possibly can adapt to frequent coding-agent failures: the mistaken file, the mistaken selector, the mistaken operate, a skipped command, an edited take a look at, a damaged route, a missed edge case, or a repair that passes one test however fails the person’s precise request.

The developer downside is broader than UI

Right here is the sort of request many builders now give to Codex, Claude Code, Cursor, or one other coding agent:

Repair the button within the pricing card. It overflows on cellular.

That immediate sounds clear to a human. It’s not sufficient proof for an agent.

The identical assessment downside seems outdoors person interface work:

• “Repair the checkout bug” can change the mistaken cost department.
  The agent would possibly edit the PayPal circulation when the bug is definitely within the Stripe circulation, or patch visitor checkout when the failure is in logged in checkout.
• “Make this take a look at move” can weaken the assertion as a substitute of fixing the conduct.
  As a substitute of fixing the damaged code, the agent would possibly change the take a look at so it checks much less. The take a look at passes, however the true bug stays.
• “Replace the API response” can edit a kind however miss the handler.
  The agent would possibly replace a TypeScript sort or schema, however overlook to vary the backend operate that truly returns the response.
• “Repair auth redirect” can patch the frontend whereas the backend nonetheless returns the mistaken standing.
  The agent would possibly change React routing, however the true difficulty is the server returning 401 as a substitute of 302, or lacking a session cookie.
• “Clear up this operate” can change conduct whereas trying like a refactor.
  A refactor ought to protect conduct. The agent would possibly rename, reorder, or simplify code in a approach that adjustments what the operate does.

The query is identical in every case: what did the mannequin ask to do, what code truly modified, and what test proved the person’s grievance was mounted?

The agent nonetheless has to reply a number of questions:

• Which file owns the pricing card?
• Which selector is the goal?
• Is the issue attributable to the cardboard, the grid, or the button?
• What does “cellular” imply for this test?
• Did the rendered UI change, or did solely the code change?

The UI instance retains these questions seen as a result of the browser can measure the failure.

For the general public model, put the damaged app and agent script in a companion repository. I take advantage of this repository title within the instructions under:

coding-agent-run-recorder

The goal HTML is a pricing card:

  
Starter
  
  
$19 per 30 days
  

    A centered plan for solo builders delivery consumer portals and inner instruments.
  
  
    Begin your workspace at the moment
  
  Speak to gross sales

The damaged CSS is the true bug:

.primary-action {
  width: 360px;
  padding: 0 28px;
  shade: #ffffff;
  background: #111827;
  white-space: nowrap;
}

At a 390 x 844 browser measurement, the cardboard is 354px vast. The button is 360px vast earlier than padding and sits inside a padded card, so it extends previous the suitable edge.

What the agent must show

A helpful coding agent run ought to reply 5 questions:

1. What information did the agent examine?
2. What selector or part did it determine because the goal?
3. What actual patch did it apply?
4. Did the app nonetheless construct?
5. Did the suitable test show the reported downside modified?

That is stricter than asking for a diff. A diff is the earlier than and after view of a code change. It could actually look cheap even when the web page remains to be mistaken. A screenshot can present the goal button mounted whereas hiding that one other part broke. A passing construct solely proves the app compiled. It doesn’t show the person interface nonetheless appears proper.

For coding agent work, the ultimate reply needs to be a abstract of proof, not a declare of success.

The instrument setup

The Python script wraps the mannequin’s instrument loop with a recorder. The mannequin can ask for a instrument. Python decides whether or not that instrument exists, runs the matching operate, and saves each the request and the outcome within the run log.

That wrapper is the helpful thought. If you wish to know what your mannequin did throughout a coding activity, file the mannequin’s instrument requests and the true operate outcomes round it.

The script offers the mannequin the identical sort of slim instruments you’d expose by an MCP server:

MCP issues right here as a result of it may package deal these instrument features for various agent purchasers. The core thought remains to be sensible: the mannequin requests a instrument, the applying validates and runs it, and the run log information each the request and the outcome.

Whether or not the instruments stay in a Python script, an MCP server, or a bigger agent framework, the assessment commonplace stays the identical.

Present instruments do a part of this

The purpose just isn’t that agent observability is unsolved. A number of instruments already hint mannequin calls, instrument calls, prices, latency, and errors.

Weights & Biases Weave tracks LLM calls, doc retrieval, and agent steps. LangSmith offers visibility into traces and manufacturing metrics for LLM functions. Arize Phoenix information mannequin calls, retrieval, instrument use, and customized logic. Langfuse captures LLM calls and non-LLM calls, together with instrument calls, API calls, retrieval, embeddings, classes, value, and latency.

There are additionally instruments aimed extra straight at coding brokers. Dynatrace describes AI coding agent monitoring throughout LLM requests, instrument executions, file snapshots, and diff operations. Arize has written about coding agent tracing throughout shell instructions, file edits, command executions, datasets, and experiments.

These instruments are helpful as soon as a crew needs hosted traces, dashboards, search, value monitoring, or analysis at scale. The hole right here is narrower: methods to wrap one model-driven coding activity so a developer can see the precise instrument requests, file adjustments, checks, screenshots when helpful, and failures that led to the ultimate repair.

You possibly can ship this similar run log to Weave, LangSmith, Phoenix, or Langfuse later. Step one is ensuring the run information the suitable issues.

Run the tutorial

The cleanest reader expertise is the companion GitHub repository: abduldattijo/coding-agent-run-recorder. Colab can run this with additional setup, however a repository is a greater default as a result of the tutorial makes use of Node.js, Playwright, generated screenshots, and a static app.

The repository ought to have this construction:

coding-agent-run-recorder/
  README.md
  ui_fixing_agent.py
  app/
    index.html
    types.css
    types.earlier than.css
    package deal.json
    scripts/
      build-check.mjs
      inspect-ui.mjs
  media/
  outputs/

Necessities:

• Python 3.10 or newer
• Node.js and npm
• Playwright’s Chromium browser, put in by the command under
• OPENAI_API_KEY for the model-driven run

Use a mission API key out of your shell or Secret Supervisor. Don’t paste the important thing into the script or commit it to the repository.

After cloning or downloading the companion repository, set up the Python package deal, Node dependency, and Chromium as soon as:

git clone https://github.com/abduldattijo/coding-agent-run-recorder.git
cd coding-agent-run-recorder
python3 -m pip set up -r necessities.txt
cd app
npm set up
npx playwright set up chromium

Then run the proof script from the companion repository root:

cd ..
python3 ui_fixing_agent.py --mode api

If you wish to regenerate the identical browser proof with out an API key, use replay mode:

python3 ui_fixing_agent.py --mode replay

The run creates these information:

outputs/before_dom_check.json
outputs/wrong_selector_dom_check.json
outputs/after_dom_check.json
outputs/target_patch.diff
outputs/build_output.txt
outputs/ui_fixing_agent_run_log.json
media/pricing-card-mobile-before.png
media/pricing-card-mobile-wrong-selector.png
media/pricing-card-mobile-after.png

In order for you a Colab model, use it as a secondary hyperlink with the repository cloned contained in the pocket book. The pocket book ought to nonetheless run the identical script and save the identical information. I might not make Colab the one path as a result of browser installs and saved screenshots are simpler to examine in a traditional repository.

The primary path makes use of the OpenAI software programming interface (API), so the mannequin chooses the subsequent instrument name. I used gpt-4.1-mini for the captured run proven right here. Replay mode runs the identical instrument features in the identical order once you need to regenerate the proof with out utilizing an API key. The essential factor is that each declare comes from captured run proof.

The browser test

The browser inspection makes use of Playwright locators and Playwright screenshots to open the web page at 390 x 844, find the goal card and button, and evaluate bounding packing containers.

The core test is:

const card = doc.querySelector('[data-component="pricing-card"]');
const button = doc.querySelector('[data-component="pricing-card"] .primary-action');

const cardBox = card.getBoundingClientRect();
const buttonBox = button.getBoundingClientRect();

const buttonInsideCard =
  buttonBox.left >= cardBox.left &&
  buttonBox.prime >= cardBox.prime &&
  buttonBox.proper <= cardBox.proper &&
  buttonBox.backside <= cardBox.backside;

This doesn’t show the design is gorgeous. It proves the particular failure within the immediate: the goal button not spills outdoors the cardboard on the chosen cellular viewport.

That distinction issues. The agent ought to confirm the person’s grievance earlier than it invents a broader design assessment.

Proof earlier than the repair

The primary DOM test fails:

{
  "label": "before_fix",
  "viewport": { "width": 390, "peak": 844 },
  "selector": "[data-component="pricing-card"] .primary-action",
  "cardBox": { "x": 18, "width": 354, "proper": 372 },
  "buttonBox": { "x": 41, "width": 360, "proper": 401 },
  "buttonInsideCard": false,
  "overflowRightPx": 29,
  "computed": {
    "width": "360px",
    "whiteSpace": "nowrap",
    "overflowWrap": "regular",
    "maxWidth": "none"
  }
}

The essential quantity is overflowRightPx: 29. The goal button extends 29 pixels previous the cardboard’s proper edge.

The mistaken selector try

Within the captured API run, the mannequin first requested to vary .support-link, which is close to the goal card, as a substitute of .primary-action, which is the overflowing button. The wrapper recorded that request earlier than Python utilized the patch.

That patch is saved in:

outputs/wrong_selector_patch.diff

The browser test catches the error:

{
  "label": "wrong_selector_attempt",
  "buttonInsideCard": false,
  "overflowRightPx": 29,
  "computed": {
    "width": "360px",
    "whiteSpace": "nowrap",
    "overflowWrap": "regular",
    "maxWidth": "none"
  }
}

That is the lesson. A mistaken edit can nonetheless be legitimate code. Settle for the repair solely when the goal browser test passes.

The centered patch

After restoring the baseline, the script applies the focused CSS patch to .primary-action:

 .primary-action {
-  width: 360px;
-  padding: 0 28px;
+  width: 100%;
+  max-width: 100%;
+  min-height: 48px;
+  padding: 0 18px;
   shade: #ffffff;
   background: #111827;
-  white-space: nowrap;
+  white-space: regular;
+  overflow-wrap: anyplace;
 }

This patch is deliberately slim. It doesn’t change the grid. It doesn’t change each button. It doesn’t disguise overflow on the cardboard. It fixes the goal button by making it match the cardboard width and permitting protected wrapping if the label adjustments later.

The construct output can also be saved:

> [email protected] construct
> node scripts/build-check.mjs

Construct test handed
Copied 2 information to dist/

Proof after the repair

The ultimate browser test passes:

{
  "label": "after_fix",
  "viewport": { "width": 390, "peak": 844 },
  "selector": "[data-component="pricing-card"] .primary-action",
  "cardBox": { "x": 18, "width": 354, "proper": 372 },
  "buttonBox": { "x": 41, "width": 308, "proper": 349 },
  "buttonInsideCard": true,
  "overflowRightPx": 0,
  "computed": {
    "width": "308px",
    "whiteSpace": "regular",
    "overflowWrap": "anyplace",
    "maxWidth": "100%"
  }
}

The goal measurement modified from overflowRightPx: 29 to overflowRightPx: 0.

The ultimate screenshot is beneficial, however the measurement is what makes the outcome simple to assessment. If a reviewer disagrees with the design alternative, they’ll nonetheless see that the particular overflow bug was mounted.

What the run log reveals

The total run log is saved in:

outputs/ui_fixing_agent_run_log.json

It information the whole chain:

mannequin requested restore_baseline
restore_baseline
mannequin requested list_files
list_files
mannequin requested read_file
read_file index.html
mannequin requested read_file
read_file types.css
mannequin requested inspect_dom
inspect_dom before_fix
mannequin requested apply_patch
apply_patch mistaken selector
mannequin requested inspect_dom
inspect_dom wrong_selector_attempt
mannequin requested apply_patch
apply_patch goal button
mannequin requested run_build
run_build
mannequin requested inspect_dom
inspect_dom after_fix

That’s the half many agent runs miss. With out the run log, the developer solely sees the ultimate code. With the run log, the developer can examine the mannequin’s requested actions and the operate outcomes that adopted.

If the agent learn the mistaken file, the run log exposes it. If it modified a worldwide selector, the patch exposes it. If it skipped the browser test, the lacking file exposes it. If the browser test nonetheless failed, the ultimate JSON exposes it.

How one can adapt this to your individual agent

The pricing card is barely the instance. The reusable sample is:

imprecise coding request
learn the doubtless information
determine the goal conduct
run the test that reveals the damaged state
apply a slim patch
run the construct
run the identical test once more
save the run log, diff, JSON, and screenshots

First, change the instruments. Hold read_file, apply_patch, and run_build, however change inspect_dom with the test that matches your bug. For an API bug, that may be call_endpoint. For a take a look at failure, it may be run_test. For a database change, it may be run_migration_check.

Second, change the proof. The proof ought to match the unique request. If the person requested a few button, use a browser measurement. In the event that they requested about checkout, use the cost department or response physique. In the event that they requested a few failing take a look at, hold the unique assertion and present the mounted take a look at output.

Third, hold the run log. The helpful file is at all times the identical: mannequin instrument request, operate arguments, operate outcome, patch, command output, and ultimate test.

This sample works past person interface work. An API bug would possibly evaluate standing codes and response our bodies. A database change would possibly run a migration test. A refactor would possibly run conduct checks earlier than and after the patch. The test adjustments, however the assessment file stays the identical.

The purpose just isn’t this pricing card. The purpose is to cease accepting “mounted” with out proof.

Sources

• Playwright documentation: Browser automation, locators, screenshots, and viewport checks.
• Mannequin Context Protocol documentation: Device boundaries and server/consumer packaging for agent-accessible instruments.
• OpenAI operate calling and instruments documentation: Device schema and tool-call loop ideas.
• OpenAI Brokers SDK documentation: Agent/instrument orchestration patterns for manufacturing programs.
• Weights & Biases Weave documentation: Present observability and analysis for brokers and LLM functions.
• LangSmith observability documentation: Present hosted tracing and observability for LLM functions.
• Arize Phoenix documentation: Present tracing for mannequin calls, retrieval, instrument use, and customized logic.
• Langfuse documentation: Present tracing for LLM calls, instrument calls, retrieval, value, latency, and classes.
• Dynatrace AI coding agent monitoring: Present monitoring for coding agent traces, instrument executions, file snapshots, and diff operations.
• Arize coding agent tracing: Present tracing and analysis workflow for coding agent runs.
• R. Torkar et al., “Necessities Traceability: A Systematic Evaluation and Trade Case Examine,” Worldwide Journal of Software program Engineering and Information Engineering, 2012: Software program engineering background on tracing a requirement from origin to success.
• coding-agent-run-recorder companion repository: Full runnable code, generated outputs, screenshots, and run log for this text.